LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


Learning gains from the KinderTEK iPad math program: 
Does timing of a preventative intervention matter? 


Mari Strand Cary 


Center on Teaching and Learning, University of Oregon, Eugene, 
OR 


Patrick Kennedy 


Center on Teaching and Learning, University of Oregon, Eugene, 
OR 


Lina Shanley 


Center on Teaching and Learning, University of Oregon, Eugene, 
OR 


Ben Clarke 


Center on Teaching and Learning, University of Oregon, Eugene, 
OR 


Strand Cary, M., Kennedy, P. C., Shanley, L., & Clarke, B. (June 5, 2020,). Learning Gains 
From the KinderTEK® iPad Math Program: Does Timing of a Preventative Intervention Matter? 
Journal of Special Education Technology, Advance online publication. 
https://doi.org/10.1177/0162643420928336 


Grant/Contract Number:]R324A110286 


Grant Funding Center: US Department of Education; Institute of Education Sciences NCSER 


LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


Abstract 


A quasi-experimental design in six kindergarten classrooms (n = 
123 students) was used to study the effects of the KinderTEK 1iPad- 
based math program on the math achievement of students in 
general education classrooms. Student math outcomes in three 
treatment (early start) classrooms were compared to math 
outcomes for students in three comparison (late start) classrooms. 
Results suggested that relatively brief exposure to KinderTEK 
produced gains on distal measures of early numeracy and that, on 
average, timing of intervention delivery did not impact end of year 
math outcomes. However, exploratory analyses suggested that 
earlier and longer use of KinderTEK may have provided a benefit 
for students most at risk in math. The utility of quasi-experimental 
studies within an overarching research program and implications 
for the adoption of technology-based math programs in 
kindergarten classrooms are discussed. 
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Educators and policy makers have been troubled by the poor math performance of U.S. 
students for more than 50 years (National Commission on Excellence in Education, 1983; 
National Mathematics Advisory Panel, 2008; National Research Council, 2001; Senate Bill 
3187, 1958; The White House, Office of the Press Secretary, 2019). Rigorous academic 
standards (e.g., Common Core State Standards Initiative, 2010) were introduced in an attempt to 
dramatically improve student math performance. Outcomes on recent national and international 
tests reveal that the desired effects have not yet materialized, particularly for students with 
disabilities, those with economic disadvantages, English learners, and many minorities 
(Bachman, Votruba-Drzal, El Nokali, & Castle Heatly, 2015; Kainz; 2019; McFarland, et. al, 
2019; NCES, 2019a; NCES, 2019b; Provasnik, et al, 2016; Schleicher, 2019). This suggests that 
many students are not benefiting from current instruction to the extent needed to succeed. The 
consequences of such under-achievement affect not only individual students throughout school 
and their lives (Gaertner, Kim, Desjardins & McLarty 2014), but also the larger society. As one 
example, students who perform poorly in math or exhibit negative beliefs about their math 
abilities are less likely to be interested in or pursue STEM careers (e.g., Huang, Zhang & 
Hudson, 2019; Holmes, Gore, Smith & Lloyd, 2018; Seo, Shen & Alfaro, 2019). This reduces 
the nation’s capacity to engage in STEM and innovate. 

Poor math performance as late as high school can be traced to poor performance in 
kindergarten and preschool (e.g., Duncan et al., 2007; Engel, Claessens, Watts & Farkas, 2016; 
Morgan, Farkas, Hillemeier, & Maczuga, 2016; Morgan, Farkas, & Wu, 2009; Watts, Duncan, 
Siegler & Davis-Kean, 2016). Burchinal et al. (2019), Pace et al. (2019), and others have found 
early math achievement to be the best predictor of later math performance. Consequently, filling 


in math knowledge gaps when students first enter school and giving students a strong start in 
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math is key to students’ later success. 

This can be accomplished by fully utilizing core math instruction time in kindergarten 
classrooms in at least three ways. First, educators can make the core lessons beneficial to all 
students by embedding instructional design principles found to be effective with students 
struggling with math (Clarke et al, 2011a; Doabler, et al, 2012). These principles include 
focusing on core concepts and visual models, carefully sequencing instruction to ensure 
prerequisite skills are learned first; demonstrating problem solving strategies and procedures; 
providing numerous, scaffolded opportunities to respond; and providing immediate academic 
feedback. Second, educators should instruct students in critical conceptual aspects of math, as 
well as build procedural knowledge (Bachman et al, 2015). For example, when teaching two 
digit numbers, teachers can go beyond teaching students to “count on” from an anchor number 
like ten or twenty. Teachers can demonstrate that in a base ten system, ones are grouped into 
groups of ten and multi-digit numbers are composed of some number of tens and some ones. 
These concepts are critical to support later learning about larger numbers, place value, and 
number operations (e.g., they help students grasp that tens can be grouped into groups of 
hundreds when students regroup in multi-digit addition; they increase the saliency of procedural 
aspects of number operations). Third, educators can adapt their instruction to meet individual 
student needs and preferences by altering variables like content and pacing, presentation, when 
and how students respond, how much support is provided, and the specificity and timing of 
feedback. Parsons et al. (2018) and van Geel et al. (2019) provide reviews of many of the ways 
in which teachers can adapt (1.e., individualize or differentiate) instruction within the general 
education classroom. However, consistently providing differentiated learning experiences to 


large groups of students is no easy task; it is one that few educators fully embody (Parsons et al., 
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2018). 
Technology offers a potential, though challenging, solution 

Technology offers one way for teachers to more fully utilize math instructional time. 
Individual learning technologies can be designed to implement all three practices just described, 
and more (Outhwaite, Faulder, Gulliford & Pitchford, 2019). Educators, researchers and leaders 
of educational associations recognize technology’s potential to motivate and engage students in 
math, assess students’ math knowledge, and simultaneously provide instruction at different 
levels and paces to different students (ASCD, 2011; Foster, Anthony, Clements, Sarama & 
Williams, 2016; Hafler, Major & Hennessy, 2016; Higgens, Hiscroft-D’Angelo & Crawford, 
2019; Ninaus, Kiili, McMullen & Moeller, 2017; Scherer, 2011). However, many technology- 
based early elementary math programs do not incorporate evidence-based instructional design 
principles, tend to favor basic facts and procedures over conceptual knowledge, and vary in 
whether—and the degree to which—they offer differentiated learning experiences. 

Beyond these issues, challenges abound when considering using technology use with 
elementary-aged children. These challenges include lack of technical infrastructure and support 
(e.g., poor Wi-Fi, limited technical support, poorly-planned technology roll-outs, insufficient 
training, outdated software or equipment), reluctance of teachers to incorporate technology 
(whether due to interest, skill, available training, or time), mismatches between tools’ content, 
depth, breadth, and approach (e.g., poor alignment with mandated curricula and student needs), 
feasibility and fit of technology with a setting and age group, and the rapidly changing 
technology landscape (Foster, Anthony, Clements, Sarama & Williams, 2016; Hawkins, Collins, 
Hernan & Flowers, 2017; Mac Callum, Jeffrey & Kinshuk, 2014). 


Thus, to develop technology that fully supports educators and students, developers must 
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(a) think like expert teachers as they choose and design content and approaches to differentiation, 
and (b) prioritize product feasibility and effectiveness. There are good examples of this in the 
non-technology sphere (e.g., the Moving Up! Mathematics series, Early Numeracy Intervention 
L1) and technology-based examples are starting to emerge (see Kiru, Doabler, Sorrells & Cooc, 
2017 for review). Educators can also find guidance in identifying and choosing technology- 
delivered evidence-based programs (EBPs) to fit their particular contexts (e.g., Doabler, et al., 
2018; Hawkins, Collins, Hernan & Flowers, 2017; National Center for Intensive Intervention, 
n.d.; Nelson et al., 2016; WWC, n.d.). As will be described next, our research and development 
team has produced such an EBP to provide differentiated early math instruction to young 
students. 
KinderTEK® 

Overview and intended uses. KinderTEK® is an iPad app that helps students develop, 
maintain, and become fluent in critical early math concepts and skills via an engaging learning 
environment (Author, 2015; Author, 2020; Author, n.d.). It includes 51 instructional lessons 
aligned with standards from three of the five Common Core State Standards in mathematics 
(CCSS-M) kindergarten domains and was developed through a public/private partnership and 
three federally funded grants. KinderTEK developers adhered to an iterative design process that 
was guided by expertise from educators specializing in math instruction, special educators, and 
game developers. 

The team conceptualized KinderTEK primarily as an individualized, prevention-oriented 
intervention for students with or at risk for math difficulties. Explicit instructional design 
features found to be particularly effective for struggling students lie at the heart of KinderTEK 


lessons. As the app delivers instruction and practice opportunities, KinderTEK goes beyond 
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group-level best practices by differentiating instruction for each student. Moment-by-moment, 
the app determines what to do next for a student based on that student’s prior interactions with 
the program. Such differentiation is recommended by the Office of Educational Technology 
(2010) and mirrors what highly effective teachers do in the classroom, particularly during one- 
on-one instruction (Parsons et al, 2018). 

KinderTEK was also intentionally designed to be used as an intervention by older 
students still struggling with kindergarten-level mathematics, as a practice and fluency-building 
activity for on-track kindergarten students, and as a school-readiness tool for preschoolers. Its 
architecture allows it to function as a sequenced (i.e., with a set scope and sequence) or flexible 
(i.e., students or teachers choose content) curricular supplement. It is not designed to replace the 
introduction and practice of new content during core mathematics instruction, but it is 
appropriate as a replacement for math worksheets, or as an option during math centers and small 
group instruction. 

Teachers can choose from three instructional modes. Sequenced mode presents all 
KinderTEK lessons in a prespecified sequence that fosters students’ learning of prerequisite 
content before more advanced content. KinderTEK use can be 10, 15, or 20 minutes in length. 
Each time a student uses KinderTEK, they work on (though perhaps do not complete) a 
minimum of two lessons, complete with embedded assessment and rewards. During their next 
KinderTEK use, the student will resume unfinished lessons and continue working at their own 
pace through the KinderTEK curriculum. Directed mode functions like sequenced mode except 
that teachers constrain lesson content to one or two categories (e.g., only lessons involving math 
models like ten frames and number lines; story problems; counting). Exploratory mode offers 


untimed KinderTEK use during which students choose between all lessons and reward activities 
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and work as much or as little as they desire in each. In most cases, teachers will choose one 
mode for a student to use throughout the year (e.g., at-risk students are likely to use sequenced 
mode; advanced students are likely to use exploration mode). In some cases, however, a teacher 
may have student use multiple modes (e.g., use sequenced mode during intervention time and 
directed mode in place of a worksheet to accompany specific core instruction). 

Smooth integration into classroom routines. KinderTEK is delivered through devices 
many schools already utilize for a multitude of educational purposes. It requires minimal teacher 
facilitation. Once teachers have installed KinderTEK on the iPad and entered their teacher 
credential (e.g., at the beginning of the school year), students can use KinderTEK independently 
anywhere in the school. Teachers can be confident students will have productive, engaging math 
experiences and be done with KinderTEK when the teacher has planned because the app adjusts 
instruction to match whatever time is selected (e.g., if students typically engage in math centers 
for 15 minutes, teachers can set KinderTEK to last 15 minutes). Automatic saving of student 
progress allows students to resume gameplay at a subsequent time and ensure teachers see up-to- 
date data. Reports of student-level progress and formative assessment data are aligned with 
learning goals and CCSS-M standards to facilitate instructional planning and conversations with 
other educators and parents. 

Teachers implementing KinderTEK receive support through professional development 
sessions (either in-person or online), on-demand technical assistance (through online chat, email, 
phone or in-person visits from the KinderTEK team), emails and newsletters, and on the 
KinderTEK website. Before starting KinderTEK, teachers are encouraged to discuss with 
students their classroom rules and expectations around iPad use generally (e.g., carry iPads 


carefully, use headphones, use the assigned app) and KinderTEK use specifically (e.g., 
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KinderTEK is math instruction and students need to try their best to learn). Teachers are also 
encouraged to have students watch an introductory video about KinderTEK and to walk students 
through the login process the first time. If desired, individual students can complete an in-app 
tutorial to practice listening to—and taking turns during-instruction and selecting or dragging 
objects to answer math questions and manipulate their digital scrapbook. 

Most students have a successful first KinderTEK experience with no pre-training. We 
expect this is because students are provided with printed reminders of their 3-animal passcode; 
the KinderTEK interface is simple, uncluttered, and easy-to-navigate; and instruction in 
KinderTEK includes modeling of what students are expected to do (e.g., drag an object, touch an 
object) before they are asked to do it. The most common hiccup to student success early on is 
that they treat KinderTEK like a typical app rather than as math instruction. Some students need 
to be reminded to “take turns” with the virtual guide who provides instruction during their 
KinderTEK journey so that they don’t get frustrated by trying to answer early. As students work, 
KinderTEK alerts teachers if students are struggling with a given activity, so that the teacher can 
confirm students understand the interface and what the system is asking them to do. 

Student experience. As students use KinderTEK, a virtual guide provides explicit 
demonstration and modeling of math models, strategies, and skills; scaffolded practice 
opportunities; and informative feedback. Capitalizing on the 1:1 digital environment, students’ 
progress through, and experience with, KinderTEK is based on the instructional mode, which 
customization features are active and students’ unique response patterns within the app. 

A major premise of KinderTEK is that students should, for most of their instructional 
time, work on material that is challenging to them. For some students, that is KinderTEK lesson 


1, but for others, challenge comes during lesson 3, lesson 10 or lesson 22. Thus, KinderTEK 


Page 9 of 49 


LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


administers a brief pretest before each lesson to determine whether the student needs to complete 
that lesson or has the prerequisite skills to move on to more advanced content (i.e., the next 
lesson’s pretest). If a student passes the pretest, that activity is marked for later review and the 
student moves on to the next activity. If they do not pass, the student enters the activity’s 
instructional phases. During instruction, the number and content of math items, scaffolding and 
feedback are dynamically adjusted in response to student performance. A student struggling with 
a skill (e.g., counting objects) may be asked to solve the problem in a scaffolded way (e.g., by 
touching to count each butterfly before selecting the answer), whereas a student who has already 
grasped it will be allowed to select the answer immediately. Whether students complete an 
activity in minutes or across multiple KinderTEK sessions, students will eventually take an 
activity posttest. If they pass it, a new activity will be unlocked. As noted above, in timed 
instruction modes, the app ensures that students encounter a mix of instruction, review and 
rewards each time they use KinderTEK. 

Instruction, rewards, and interface navigation are supported by clear, appropriately paced 
audio using age-appropriate vocabulary with which children are already familiar or have been 
explicitly taught through KinderTEK. Further support comes from, a handful of symbols and 
icons recognizable to students (e.g., common math symbols; arrows to move forward and back; 
mini-replicas of KinderTEK features like the activity map, scrapbooks and puzzle) and visual 
models and prompts (e.g., onscreen action illustrates concepts and directs students’ attention). 

To enhance motivation and engagement, teachers can customize many instructional 
settings to fit each student’s needs and preferences. To name some examples, sessions can be 10, 
15 or 20 minutes long or open-ended; rewards can be tied more heavily to perseverance or 


mastery or be balanced; time in the scrapbook and activity center can come at the end of the 
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session or also at the midpoint. As well, attention supports like onscreen and audible indicators, 
progress bars, and countdown timers can be adjusted and students can be given more or less 
control over their experience (e.g., activity choice, pause option, replay options). 

Current study’s aims and research questions 

Throughout the development of KinderTEK, we evaluated the feasibility and 
effectiveness of specific features and components via stakeholder user-tests, focus groups, and 
small learning trials. As complete prototypes and the full product emerged, our emphasis shifted 
to examining the feasibility of the program as a whole and its effectiveness at improving student 
outcomes using larger experimental and quasi-experimental studies. Results of such studies will 
be useful to practitioners deciding which interventions to implement, with whom and in which 
ways (Nelson & McMaster, 2019). Given the myriad ways KinderTEK can be used with 
different student populations and in different contexts, multiple studies of feasibility are critical. 
The quasi-experimental study we present here focused on one such feasibility question: To what 
extent does the timing of KinderTEK use in general education classrooms affect student 
outcomes? 

Our first aim was to examine how limited, early exposure to KinderTEK impacted 
student performance on a distal math measure assessing early numeracy skills (i.e., number 
identification, magnitude comparison, missing number). We compared use of the program in the 
fall to business-as-usual (BAU) math instruction to answer our first research question: To what 
extent did fall-to-winter gains for students who used KinderTEK during the first half of the 
school year differ from those of their comparison peers who had not yet used KinderTEK? We 
hypothesized that treatment students using KinderTEK would make greater fall-to-winter gains 


on a standardized test of early numeracy than their comparison group peers. 
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Our second aim was to contrast two potential uses of the program: (a) throughout 
kindergarten (the early start treatment condition) and (b) in the second half of the year (the late 
start comparison condition). Our second question was: Did starting KinderTEK earlier in the 
year result in different fall-to-spring gains than starting later in the year? We hypothesized that 
early start students would gain at least as much on the standardized test of early numeracy as 
their /ate start peers from fall to spring. Given KinderTEK’s differentiated approach and 
comparison group students’ greater development and exposure to classroom instruction before 
they encountered KinderTEK, it was not clear whether we should expect additional months of 
use to result in greater outcomes for early start students. 

Our third aim was to explore group differences in effects for students by initial math skill. 
Hypotheses for these analyses were informed by three factors. First, KinderTEK was designed 
primarily to address the needs of struggling students. Consequently, initial KinderTEK lessons 
focus on foundational skills related to number modeling, identifying, and sequencing numbers. 
Second, we expected lower performing students to pretest into the earlier KinderTEK lessons. 
They would jump right into instruction. Third, we expected higher performing students to test 
out of KinderTEK’s initial lessons (i.e., spend days and weeks completing pretests before 
encountering challenging material). Given these factors, we hypothesized lower performing early 
start students would outperform their late start comparison peers in the winter, and that exposure 
to those early foundational skills would continue to provide an advantage such that early start 
students’ spring scores would also be higher. We hypothesized higher performing early start 
students would show limited gains initially (1.e., perform similarly to their late start comparison 


peers in the winter). As they began engaging in KinderTEK instruction, we expected higher 
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performing early start students to gain skills and outperform their /ate start comparison peers in 
the spring. 
Method 
Design 
This study was part of a larger product development study included in our Office of 
Special Education Programs (OSEP)-funded grant. That larger study required all participants to 
use KinderTEK starting in February and represented the first sustained implementation of the 
full KinderTEK product. As that study ramped up, we had the opportunity to implement the 
quasi-experimental study reported here. Namely, we compared student math performance of 
three classes that started KinderTEK relatively early in the school year (Treatment; early start) to 
the performance of three classes that did not start using KinderTEK until later in the year 
(Comparison; late start). As shown in Figure 1, assessments were administered in the fall, before 
any classrooms used KinderTEK (T1); midway through the year, after the treatment students had 
used KinderTEK for several weeks, but the comparison group had not (T2); and at the end of the 
school year when both the early start and late start groups had used KinderTEK (T3). 
<< Figure | here>> 
Setting and participants 
The study was conducted in six kindergarten classrooms in two elementary schools in a 
single Pacific Northwest school district. One school had four kindergarten classes, and the other 
had two. Teachers at both schools were ready for the larger implementation study earlier than 
expected, thus conditions were ripe for this quasi-experimental study. Classrooms whose 
teachers were ready to begin KinderTEK early (two in the larger school and one in the smaller 


school) were considered treatment (early start) classrooms. The remaining classrooms (again, 
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two in the larger school and one in the smaller school) served as comparison (late start) 
classrooms. All classrooms used the same core math curriculum, and, within school, teachers had 
grade-level planning meetings and introduced lessons at the same pace using the same scripting, 
workbooks, worksheets and resources. Following approved Institute Review Board procedures, 
all kindergarten students in the six classrooms (7 = 135) were invited to participate, and 129 
consented and completed the pretest assessment. Six students in the late start condition were 
excluded from this study because they also participated in a separate single subjects study that 
gave them access to KinderTEK content earlier than their /ate start peers, resulting in a study 
sample of 123 students, 64 in the early start condition and 59 in the late start condition. 
Demographic data were available for all but one student: 50.4% were female, 15.4% were 
English learners, 10.6% were eligible for special education services, 57.7% were white, 26.8% 
were Hispanic, and 12.2% were of two or more race/ethnicity categories. 
Intervention 

KinderTEK (Authors, 2016) was the intervention during this study. Teachers were asked 
to implement KinderTEK in sequenced mode with all their students. We stated a usage goal of 
15 minutes per day, 3 days per week, but did not enforce adherence to that schedule. Students 
were expected to use KinderTEK individually at their desks or at a station in the classroom. 
Teachers had the ability to change settings (e.g., mode, minutes, on-screen student supports and 
reward timing), but were encouraged to use the default sequenced mode throughout the study. 

Professional development and ongoing support. During this study, KinderTEK 
professional development (PD) was provided primarily in person. Thus, prior to implementing 
KinderTEK in their classrooms, teachers were provided with a full day of in-person PD. The PD 


was led by the research team and program authors and was intentionally structured so that as 
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teachers shared their instructional goals and challenges related to kindergarten math, the 
presenters introduced and explained how the instructional principles and architecture underlying 
KinderTEK would help them meet students’ needs. Educators also played KinderTEK to better 
understand how students could experience the game, practiced performing common teacher tasks 
related to the program (e.g., how to add students and review progress reports), and were given 
time to plan how they would integrate KinderTEK into their classroom instruction. Research 
team members provided in-class support to teachers during their first days of KinderTEK. As 
well, teachers had access to online KinderTEK supports (e.g., how-to guides, brief videos about 
using reports) and on-demand support (in person and virtually) from the research team 
throughout the study. 

Treatment condition. Teachers in early start classrooms were asked to supplement their 
regular math instruction by having all students use KinderTEK starting in the fall. Two of the 
three teachers in early start classrooms received the PD in early November and began using 
KinderTEK in mid-November. The other teacher had completed the same PD and implemented 
KinderTEK for a short time at the end of the previous school year, so began implementing 
KinderTEK in mid-October, before the PD for the other early start teachers took place. Fidelity 
of implementation and dosage details for the analytic sample are provided in the results section. 

Comparison condition. Teachers in /ate start classrooms proceeded with their regularly 
planned math instruction in the fall. In mid-February, comparison teachers (and additional 
teachers taking part in the larger KinderTEK study) received the same PD that treatment teachers 
had received in the fall and their students began using KinderTEK. All app content and supports 


provided during comparison students’ use of the app was identical to that available to students in 
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the treatment condition. Fidelity of implementation and dosage details for the analytic sample are 
provided in the results section. 
Measures 

Students completed the Assessing Student Proficiency in Early Number Sense (ASPENS; 
Clarke, Rolfhus, Dimino & Gersten, 2011b). The ASPENS assessment is a standardized, 
individually administered test of early number sense consisting of three timed subtests, each 
taking 1-2 minutes. In kindergarten, ASPENS assesses students’ ability to say the names of 
numerals (Number Identification), compare two numerals and determine which is greater 
(Magnitude Comparison), and identify the missing numeral in a string of three numerals 
(Missing Number). These skills are the focus of earlier KinderTEK lessons and certainly 
kindergarten core instruction; thus, the assessment is well-aligned with the content to which 
students were exposed during this study. Individual subtest scores are weighted to form an 
overall ASPENS composite score. The ASPENS authors report test-retest reliability ranging 
from .71 to .90 and concurrent and predictive validity with the TerraNova Third Edition as 
between .57 to .63 (Clarke, Gersten, Dimino, & Rolfhus, 2012). The ASPENS Administrator’s 
Manual also provides normative benchmark information based on a sample of 353 kindergarten 
students in six schools from four districts in Ohio and California (Clarke et al., 2012). The 
ASPENS measures were administered at Tl (as a pretest measure and to categorize students by 
level of math risk), T2 (as an interim measure) and T3 (as a posttest measure). 
Statistical Analysis 

As a first step in the analytic process, each of the three ASPENS subtests and the 
ASPENS composite score were summarized descriptively (see Table 1) and evaluated for 


baseline equivalence. No statistically significant differences were found between the two 
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conditions on any of the measures at T1. Gain scores based on differences from T1 to T2 and 
from T1 to T3 were calculated and evaluated for assumptions of normality using the R statistical 
programming language (R Core Team, 2018) package compareGroups (Subirana, Sanz, & Vila, 
2014). For normally distributed measures, differences in gains were evaluated using an analysis 
of variance (ANOVA) approach. For non-normally distributed measures, differences were 
evaluated using the Kruskal-Wallis Rank Sum Test. To account for the inclusion of multiple tests 
on the same sample, all p-values were also adjusted using the Benjamini and Hochberg (1995) 
correction for false discovery. Estimates of effect size were computed using Hedges' g (1981), 
which pools variances on the assumption of equal population variances, using n - | for each 
sample. Statistical significance was specified a priori at p < .05. 
<< Table 1 about here>> 

Follow-up exploratory analyses were then conducted to investigate whether there were 
group differences in gains by initial skill level, as measured by ASPENS Composite scores at T1. 
Gains were summarized descriptively by ASPENS performance level, and effect sizes of group 
differences were calculated using Hedges' g. 

Results 

Attrition and Missing Data 

Across conditions, attrition and missing data impacted 7% of students at T2 and 14% of 
students at T3. Specifically, among students who were assessed at T1, ASPENS data were 
unavailable at T2 for six students and at T3 for 14 students, and three additional students had no 
KinderTEK use at any point in the year. Thus, our analyzed sample sizes were 114 students at T2 
(93% of participants) and 106 students at T3 (86%). By condition, six treatment students (9.4%) 


and three comparison students (5.1%) were excluded from analyses of T2 data, and nine 


Page 17 of 49 


LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


treatment students (14.1%) and eight comparison students (13.6%) were excluded from analyses 
of T3 data. These rates of attrition fall within the acceptable range for both overall and 
differential attrition as defined by the What Works Clearinghouse guidelines for assessing 
attrition bias (Institute of Education Sciences, 2014). 
Levels of students’ mathematical risk 

Most students are not identified as having learning disabilities before or during 
kindergarten. Thus, performance on the ASPENS assessment at Tl was useful not only as a 
continuous variable utilized in primary outcomes analyses, but also as a categorical variable used 
in our exploratory analyses. ASPENS provides three performance level categories, based on 
students’ composite score (Clarke et al., 2012). Students at benchmark are performing at the 
level expected and are considered on track; those in the strategic zone are at risk for being below 
benchmark at end of year; and those in the intensive zone are clearly not on track to reach end- 
of-year benchmarks (1.e., they are performing below the level expected). The two conditions had 
similar proportions of students in each ASPENS performance level. The early start condition 
included 18 students (31%) identified as needing intensive support, 26 students (45%) identified 
as needing strategic support, and 14 students (24%) identified as needing benchmark support. 
The late start condition included 20 students (37%) identified as needing intensive support, 22 
students (41%) identified as needing strategic support, and 12 students (22%) identified as 
needing benchmark support. 
Fidelity of Implementation and Usage 

As this was the first sustained, full-scale implementation of the complete KinderTEK 
product by teachers, fidelity of implementation was not a major focus of this or the parent study. 


Nonetheless, research staff regularly visited implementing classrooms to identify major 
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implementation roadblocks and contextual characteristics that could influence further program 
revisions. KinderTEK gameplay data (1.e., log data) were collected to document dosage. 

Treatment condition. During classroom visits (5-6 per treatment classroom), 100% of 
students experienced KinderTEK individually, with headphones, at their desks or at a station in 
the classroom. Log data confirmed that students in the treatment condition began using 
KinderTEK when requested (median start date = November 17", min = October 17", max = 
November 21‘), and indicated that 75% of students engaged in instructional activities for 
between 10 and 15 minutes each session, confirming that teachers typically set students’ game 
play to last for 15-minutes, as requested. Most students experienced sequenced mode, as 
intended. As Table 2 shows, students in one class (33% of treatment students) used a 
combination of directed and exploration mode for several days in the latter part of the year. 

Usage was far less than suggested and far more variable across classes and students than 
desired. Students in treatment classrooms used KinderTEK on an average of 28 days (SD = 7.58, 
min = 12, max = 39) across an average of 24 weeks (SD = 2.99, min = 15, max = 39). Not 
surprisingly, given this dosage, students tended not to progress through the entire KinderTEK 
curriculum. They encountered an average of 23 different KinderTEK lessons (SD = 8.85, min = 
9, max = 47), which represents just under half of the KinderTEK lessons (1.e., students 
encountered only the earliest content). As these standard deviations and ranges illustrate, 
students did show substantial variability in usage, but notably, even those students who had the 
highest usage encountered KinderTEK less than twice per week, and only three students 
encountered at least 80% of KinderTEK lessons. 


<<Table 2 about here>> 
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During the PD and throughout implementation, we encouraged teachers to monitor 
students’ use of KinderTEK and review student reports. Monitoring could help teachers 
determine if students (a) were working productively or if they could benefit from technical 
assistance (adjusting headphones; replacing iPads with dead batteries) or KinderTEK setting 
adjustments (e.g., shorten KinderTEK sessions, increase the reward frequency, change modes, 
etc.); (b) needed assistance staying on task (in which case teachers could turn on the iPad’s 
guided access feature); or (c) could use teacher help with a skill with which they were 
continually struggling. Log and observation data indicated that treatment teachers provided 
technical assistance, but seldom changed the default settings. 

Comparison condition. A review of KinderTEK logs confirm there was no treatment 
diffusion: KinderTEK was not accessed by students in the comparison group until after the 
winter break, as planned. The median KinderTEK start date for students in the comparison 
condition was February 15 (min = February 13", max = March 2™). As with treatment students, 
researchers noted that comparison students appropriately engaged with KinderTEK and 100% of 
students were using KinderTEK individually, with headphones, at their desks or at a station in 
the classroom. Fewer visits (1-3 per classroom) occurred in comparison classrooms because 
KinderTEK was implemented across fewer months and because visits were more difficult to 
schedule for some of these teachers. Similar to treatment students, log data revealed that 
comparison students had low and variable exposure to KinderTEK (see Table 2 for class-level 
results). Students in comparison classrooms used KinderTEK on an average of 22 days (SD = 
11.44, min = 1, max = 44) across an average of 11 weeks (SD = 2.49, min = 1, max = 15), and 
encountered an average of 22 different KinderTEK lessons (SD = 10.96, min = 2, max = 47). 


Notably, their total exposure to KinderTEK content was comparable to that of the treatment 
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group, despite starting later in the year. Log and observation data indicated that, like treatment 
students, 75% of comparison students engaged in instructional activities for between 10 and 15 
minutes per session and that, like treatment teachers, comparison teachers provided technical 
assistance, but seldom changed the default settings. 
Research Question 1 

Our first research question examined whether students using KinderTEK in the fall 
gained more between fall (T1) and winter (T2) compared to their BAU comparison peers (i.e., 
students who had not yet used KinderTEK). As shown in Table 1, the treatment group had higher 
T1 to T2 gains for all measures, suggesting that student use of KinderTEK resulted in learning 
across multiple components of early numeracy skills that transfers to a distal measure of math 
proficiency (ASPENS). As shown in Table 3, prior to applying the Benjamini and Hochberg 
(1995) correction for false discovery, the difference between conditions from T1 to T2 was 
statistically significant for the ASPENS Composite score (F(1,112) = 5.50, p = .021, Hedges' g = 
44), and for the Magnitude Comparison (Kruskal-Wallis vy? = 4.79, df = 1, p = 0.029, Hedges' g 
= .33) and Missing Number subtests (F(1,112) = 4.07, p = .046, Hedges’ g = .38). After applying 
the correction for false discovery, the p values for all four measures were smaller than the 
corresponding critical values. For all measures, reported effect sizes represents relatively small 
effects. Results for all measures are shown visually in Figure 2. 

<<Figure 2 about here>> 

Research question 2 

The second research question asked whether starting KinderTEK earlier in the year 
resulted in larger beginning (T1) to end (T3) of year gains than starting later in the year. Like the 


trends observed from T1 to T2, the early start (treatment) group had larger T1 to T3 gains for 
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three of the four measures (see Table 1), although none were statistically significant. Indeed, 
students who started using KinderTEK later in the year made greater gains from T2 to T3, such 
that they ended the year, on average, with similar skills as their peers who started using 
KinderTEK earlier (depicted in Figure 2). 

Differences by skill level 

Exploratory results by risk category are based on the descriptive statistics reported in 
Table 1 and are depicted visually in Figure 3. Here, we describe the results for the ASPENS 
Composite score, but each subtest showed similar patterns. As predicted, differences in gains 
from T1 to T2 were most pronounced for students in the intensive category (mean gain of 33.33 
for the intensive treatment group compared to a mean gain of 18.56 for the intensive comparison 
group, Hedges’ g = .62, a medium effect). In contrast, students in the strategic category showed 
nearly identical gains from T1 to T2 in both conditions (mean gain of 36.62 for the strategic 
treatment group compared to a mean gain of 38.82 for the strategic comparison group, Hedges' g 
= -.08, a negligible effect). Unexpectedly, however, students in the benchmark category showed 
a trajectory from T1 to T2 that was similar to—but more pronounced than—that of students in the 
intensive category (mean gain of 37.33 for the benchmark treatment group compared to a mean 
gain of 3.73 for the benchmark comparison group, Hedges' g = 1.23, a large effect). 

We also examined gains from T1 to T3, a time span that reflects similar overall levels of 
KinderTEK exposure, but concentrated in the later part of the school year for late start students. 
Over the course of the year, students in the early start intensive category made greater gains—and 
ended the year higher—than their late start intensive peers (mean gain of 65.82 for the intensive 
treatment group compared to a mean gain of 49.35 for the intensive comparison group, Hedges' g 


= .44, a small effect), with nearly all of that difference occurring between T1 and T2. In contrast, 
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students in the early start strategic and early start benchmark categories exhibited slightly lower 
gains from T1 to T3 than their comparison peers (mean gain of 65.84 for the strategic treatment 
group compared to a mean gain of 70.07 for the strategic comparison group, Hedges' g = -.17, a 
negligible effect; mean gain of 46.45 for the benchmark treatment group compared to a mean 
gain of 52.48 for the benchmark comparison group, Hedges' g = -.16, a negligible effect). 
<<Figure 3 about here>> 
Discussion 

This quasi-experimental study was designed to evaluate the effects of the KinderTEK 
iPad-based math program on the math achievement of students in general education classrooms. 
Student math performance in three treatment (early start) classrooms was compared to student 
math performance in three comparison (late start) classrooms to determine (a) whether brief 
exposure to KinderTEK meaningfully impacted student math achievement, and (b) whether 
starting KinderTEK in the fall and continuing through the school year conveyed an advantage 
over using KinderTEK only in the second half of the year. 

Findings related to the first research question suggest that kindergarten students do 
benefit from using KinderTEK relatively early in the school year. Given that the concept of 
numerical magnitude is a major focus of early KinderTEK lessons, the finding that treatment 
students outperformed their peers on the ASPENS magnitude comparison measure by a third of a 
standard deviation is promising because it suggests that KinderTEK impacts a major skill it was 
designed to affect and ostensibly provides a strong foundation for later math concepts. It is 
noteworthy that even relatively brief KinderTEK use is also linked to gains of nearly half a 


standard deviation on a multi-component distal math measure (i.e., the ASPENS Composite). 
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These findings suggest KinderTEK is helping students learn relevant material in a meaningful 
way, beyond what is being taught in the core curriculum. 

Analyses related to the second research question revealed that both early and late start 
students made comparable gains on the multi-component distal math measure by the end of the 
school year. This suggests that KinderTEK can be successfully implemented in more than one 
way (i.e., early vs. later in the school year). Similar dosage has similar benefits regardless of 
timing. KinderTEK appears to be flexible enough to accommodate educators wishing to take 
additional time to identify kindergarten students who may benefit from supplementary math 
instruction. That students starting KinderTEK later in the year ended year on par with their early 
start peers may be accounted for by patterns of student use during the study. Dosage was much 
lower than expected and it is unclear whether both groups’ gains would have been different if 
usage had been higher. Indeed, as Pianta, et al. (2019) note, “...rigorous instructional content is 
not sufficient, in and of itself, to produce improvements in child outcomes. Children also have to 
be exposed to a sufficient dosage of such content through instruction (p2).” Alternatively (and 
described next), KinderTEK’s design may be at play. 

Recall that each KinderTEK lesson begins with a pretest that determines whether a 
student will complete a particular lesson’s instructional phase. Students with a good grasp of 
number sense typically spend many days of KinderTEK experiencing and testing out of material 
they already know. Given that students in both conditions were (a) getting older, and (b) 
receiving core math instruction throughout the study, we expected late start students to know 


more math when they began using KinderTEK in the winter than their early start peers did when 


they began KinderTEK in the fall. This was certainly true; /ate start students had an average 


winter (T2) ASPENS composite score of 21.3, compared to the average fall (T1) ASPENS 
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composite score of 6.28 for early start students. Thus, we expected late start students to test out 
of more KinderTEK material and to move more rapidly through the KinderTEK curricular 
sequence than did their peers whose performance indicated that they needed the instruction 
offered in those lessons. This pattern confirms that KinderTEK functioned as intended (i.e., it 
systematically adjusted instruction based on student performance, allowing for a dynamic 
instruction approach) and the results of this study supplement a growing body of research on 
delivery models that modify instruction based on student response and observe positive learning 
gains (Coyne et al., 2013; Al Otaiba et al., 2014). 

Exploratory analyses of gains by initial risk suggest students who are most at risk for 
math difficulty benefit from using KinderTEK earlier in the year (i.e., using KinderTEK as a 
preventative intervention). Students in the intensive category at T1 who used KinderTEK earlier 
in the year made greater gains by the end of the year than their intensive peers who started using 
KinderTEK later in the year (see Figure 3). This may be because these students had time to 
experience and master more of the KinderTEK curriculum and thus learned more math from it 
than did their peers. It may be because early use of KinderTEK helped these students gain more 
from their core classroom instruction than their peers. Regardless, this finding maps well onto 
the reading Response-to-Intervention (RTI) literature that indicates providing immediate 
intervention (i.e., Tier 2 or 3 instruction) to students who perform poorly on a screening measure 
results in both immediate and cumulative effects compared to intervening only after evaluating 
students’ response to Tier | instruction (Connor, Morrison, Fishman, Schatschneider & 
Underwood, 2007; Al Otaiba et al., 2014). In a more recent study, researchers found a similar 
effect in kindergarten mathematics (e.g., Shanley et al., 2018), suggesting that if the probability 


of a student’s response to instruction can be accurately predicted from screening scores, it may 


Page 25 of 49 


LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


be more productive and efficient to provide increasingly intensive interventions earlier in the 
school year rather than waiting to confirm that Tier | instruction is not sufficient. 

Risk-subgroup analyses also showed that strategic students who used KinderTEK early 
in the year and those who started later in the year had nearly identical patterns of performance on 
the ASPENS assessment at all three timepoints. It may be that teachers are providing core class 
instruction particularly well-aligned with this (the largest) group’s needs, such that any 
additional effects of KinderTEK are negligible. Given the usage results, it is also quite possible 
that—regardless of start date-to make meaningful progress through KinderTEK and reach more 
advanced content and fluency, strategic students must use KinderTEK more often than they did 
during this study. 

Benchmark students who used KinderTEK early in the year made greater gains than their 
BAU peers between T1 and T2. We hypothesize this is because the KinderTEK curriculum 
provided opportunities to review and build fluency on material benchmark early start students 
already knew as well as opportunities to learn material in KinderTEK that was not yet being 
covered in the core curriculum. Between T2 and T3, their performance leveled out while their 
late start peers climbed, such that by the end of the school year, both groups were performing 
similarly. This may be because core math instruction in the second half of the year leveled the 
playing field (such that all benchmark students now had substantial experience with skills tested 
through ASPENS), because of diminished interest or motivation of benchmark early start 
students to play KinderTEK, or because of high interest and motivation to learn by benchmark 
late start students. 


Limitations and Future Research 


Page 26 of 49 


LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


Individualized, differentiated learning environments such as KinderTEK present 
challenges to traditional or small-scale research designs. Such programs can be used in many 
more ways than can be feasibly tested in any single study and dosage can be defined in many 
ways (e.g., days of use, minutes of use, content coverage, time spent per skill, and more). In fact, 
this is a constant tension related to evaluating flexible, individualized instructional systems. The 
power of such systems is likely to come from their ability to adapt to each students’ learning 
needs and, in the case of KinderTEK, to teachers’ classroom goals and practices. Yet obtaining 
large enough sample sizes and documentation of such variability represents a massive 
undertaking out of reach of many research teams. 

More rigorous, large-scale efficacy studies of KinderTEK are underway, but the study 
presented here was quasi-experimental in nature and involved a relatively small number of 
participants in only six classrooms. As depicted in Figure 1, we examined KinderTEK use 
starting earlier in the year compared to use of KinderTEK only later in the year. Although there 
were measureable differences on a standardized assessment, this design paints only a partial 
picture into how KinderTEK would look (and have effects) under authentic conditions. This was 
most teachers’ first experience with KinderTEK, students used KinderTEK less often than 
recommended and inconsistently, and there was large variation across classes. We plan to 
conduct analyses of effects controlling for dosage (operationalized as days of use as well as 
progress and mastery in KinderTEK) to better understand this rich dataset. 

It would be interesting for future research to compare outcomes of students under these 
early start and late start conditions with those who did not use KinderTEK at any point in the 
year and to those who only used KinderTEK at the beginning of the year. Designing studies so 


that students use KinderTEK for a set number of days (e.g., 40 days of use) rather than over 
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particular spans of time (e.g., “fall’”), randomizing at the student level, and involving a much 
larger sample would also be informative, though more logistically challenging in school settings. 
Regardless of study designs, future work on this data set and those to come will explore in more 
detail the tremendous variability in data related to usage, perceptions, and math performance 
(including measures targeting a greater breadth of math concepts and skills than included on 
ASPENS) for full samples and for subgroups defined by classroom or risk status. 

Systems like KinderTEK are attractive for teachers because of the options they provide; 
as teachers get comfortable with basic implementation (e.g., throughout the school year), they 
are more interested in—and willing to—explore those options. When done with care, such 
experimentation is an advantage and speaks to how KINDERTEK can be used in different ways 
in the same classroom. As occurred in this study, it also complicates interpretations. Specifically, 
one of the three early start teachers switched all her students (1.e., a substantial portion of our 
early start sample) to directed mode for approximately the last third of their KinderTEK use. 
Modules activated by the teacher were “story problems” (6 lessons; likely not new content for 
many of the students as these lessons are typically early in the KinderTEK sequence) and 
“decomposition of teen numbers” (2 lessons; likely too advanced for many of the students as 
they are the last lessons in the KinderTEK sequence). Though the class may have benefited from 
story problem practice and exposure to the decomposition concepts, the teacher did not, as 
recommended, activate new modules when students mastered existing ones. Because of this, 
students repeated the same activities many times. We do not know how students in that class 
would have performed on the T3 assessment if they had continued to progress in KinderTEK’s 
recommended order, constantly building on their existing knowledge while being pushed to gain 


new skills. To alleviate this in the future, we have refined teacher alert systems and are 
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developing just-in-time, mini-PD modules to help teachers productively embrace KinderTEK’s 
flexibility and continue to reinforce to teachers the importance of monitoring KinderTEK student 
data to inform implementation and their own instruction. 

Given the limited research and findings for educational technology thus far (Kiru et al, 
2017; Young et al., 2012), it is imperative to investigate more deeply the conditions under which 
technology is deployed and utilized and the extent to which it is effective. Thus, during such 
larger studies, it would also be useful to study whether and to what extent preparedness, 
implementation, and engagement variables mediate or moderate student outcomes and document 
the effects of specific program improvements to inform future development projects. We have 
attempted to do this in our KinderTEK work to-date. For example, our earliest KinderTEK 
prototype was subject to usability and feasibility testing before we conducted a small scale RCT 
(Author 2018a). Since expanding the program, we have examined the effectiveness of specific 
supports and customization options (e.g., Author, 2019), the feasibility of the program for 
English learners in a summer school program (Author, 2018b), improvements in engagement and 
instructional time as a function of program improvements (Author, 2018c), and supportiveness 
and effectiveness of varying professional development approaches. We are currently conducting 
a federally funded KinderTEK efficacy study in two states. Beyond understanding whether and 
why KinderTEK is effective, by further exploring implementation data from all these studies, we 
hope to have a better understanding of what KinderTEK customization options and dosage are 
optimal and reasonable for different groups of students and contexts, to inform recommendations 
to educators. We believe comprehensive investigations like these are crucial for our team and all 


developers, to help ensure the promise of educational technologies for the classroom by more 
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effectively and efficiently developing, studying, refining, implementing and scaling up effective 
educational approaches for a range of learning objectives and unique contexts. 
Conclusion 

Practical solutions for providing all students with individualized instruction within core 
instruction are critical as schools consider how best to provide support given limited resources. 
The implementation in general education kindergarten classrooms described here approximated 
authentic use of iPad apps and supplementary math instruction in kindergarten settings. 
KinderTEK is likely to be used most consistently and for the longest period of time as a 
supplement to typical instruction for students struggling with math, but it will also serve as a 
practice tool for students on track in their math learning or to identify and fill gaps in knowledge 
for students intermittently struggling. As students progress through KinderTEK, the program 
differentiates instruction so that students only work on content for which they need instruction or 
practice. If students have mastered the concepts covered in KinderTEK, they can use the 
program to build fluency. As the results of this study show, KinderTEK can be used effectively 
at different times in general education classrooms. Even the benchmark students showed initial 
gains after using KinderTEK, suggesting they benefitted from the program’s differentiation. That 
said, some student subgroups may benefit more from starting KinderTEK earlier in the year. In 
this study, early start intensive students’ gains were more linear than those of benchmark 
students’, but they ended higher than their /ate start comparison peers, suggesting an ongoing 
benefit of engaging with KinderTEK’s differentiated instruction and content. 

Given the longitudinal development of math proficiency, it is imperative that all students 
exit kindergarten with a solid math foundation so that they can engage successfully with later 


elementary school math content (Geary, 1993; Jordan, Kaplan, & Hanich, 2002). To 
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simultaneously prevent long-term difficulties in math while potentially accelerating the learning 
of all kindergarten students, it is critical to use quality programs that meaningfully improve 
student outcomes and fully utilize available math instruction time by differentiating instruction at 


every opportunity. Technologies like KinderTEK are a promising means of achieving this goal. 
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Table 1. 


Early start vs. late start student math performance at T1, T2, and T3—overall and by ASPENS 


risk category at T1 
Measure ASPENS Start WN Tl T2 T3 Tl to T2 T1 to T3 
risk group type gains gains 
M SD M SD M SD M SD M_ SD 
ASPENS All Early 58 41.09 35.50 76.86 44.86 101.76 38.51 35.77 26.31 61.25 29.70 
Composite students 
Late 56 37.19 37.67 61.25 43.71 99.39 52.81 24.06 26.96 59.43 34.88 
Intensive Early 18 6.28 5.47 39.61 23.27 71.88 21.18 33.33 24.26 65.82 21.60 
Late 20 2.74 4.28 21.30 24.94 51.56 49.60 18.56 22.70 49.35 47.72 
Strategic Early 26 36.77 11.68 73.38 32.95 103.04 30.56 36.62 27.27 65.84 25.81 
Late 24 36.73 13.17 75.54 31.86 107.30 29.56 38.82 24.10 70.07 23.56 
Benchmark Early 14 93.89 22.52 131.21 30.01 138.38 38.56 37.33 28.69 46.45 41.28 
Late 12 95.52 28.84 99.25 37.85 148.00 39.25 3.73 23.44 52.48 29.59 
ASPENS All Early 58 6.21 7.46 13.55 9.77 18.49 9.57 8.69 7.08 12.33 8.87 
MC students 
Late 56 5.46 7.74 10.38 10.05 18.31 11.63 6.16 8.18 12.43 9.63 
Intensive Early 18 1.17) 147 5.89 4.03 11.47 6.47 5.22 4.12 10.24 6.59 
Late 20 0.15 049 2.80 5.03 8.75 10.34 2.75 5.06 8.69 10.36 
Strategic Early 26 3.62 3.13 12.15 741 19.12 7.46 7.42 7.11 15.36 7.38 
Late 24 462 3.19 13.79 8.15 20.43 8.56 9.58 7.87 15.78 8.68 
Benchmark Early 14 17.5 6.02 26.00 6.30 26.46 10.26 15.50 5.52 9.23 12.32 
Late 12 16.00 10.21 16.17 12.50 27.00 9.86 5.00 10.61 11.00 8.83 
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Measure ASPENS Start WN Tl T2 T3 Tl to T2 T1 to T3 
risk group type gains gains 
M SD M SD M SD M SD M_~ SD 
ASPENS All Early 58 4.86 4.49 9.29 5.89 11.91 5.18 4.43 3.83 7.11 4.00 
MN students 
Late 56 4.21 4.91 7.14 5.56 11.24 6.55 2.93 4.12 6.73 4.22 
Intensive Early 18 067 1.03 450 3.63 8.00 4.76 3.83 3.96 7.29 4.74 
Late 20 0.05 0.22 2.20 2.119 5.75 5.65 2.15 2.21 5.69 5.59 
Strategic Early 26 4.73 2.62 969 498 12.52 3.92 4.96 3.87 7.60 3.00 
Late 24 421 2.86 867 4.06 11.61 4.25 446 4.60 7.48 3.42 
Benchmark Early 14 10.50 3.92 14.71 4.83 15.85 4.56 4.21 3.75 5.92 3.33 
Late 12 11.17 4.34 12.33 5.73 17.83 4.90 1.17 4.75 6.67 3.47 
ASPENS All Early 58 17.41 14.49 28.72 16.39 38.20 14.41 11.31 12.10 21.13 13.44 
NI students 
Late 56 16.52 16.05 24.32 16.70 37.86 19.05 7.80 9.62 20.08 13.63 
Intensive Early 18 2.50 3.73 17.44 11.67 30.82 7.19 14.94 11.82 28.76 8.52 
Late 20 2.35 4.28 10.65 12.89 21.19 20.05 8.30 10.71 19.25 18.32 
Strategic Early 2617.85 6.94 26.58 12.84 36.76 14.37 8.73 11.16 19.24 12.70 
Late 24 17.50 12.37 28.62 13.55 41.22 10.56 11.12 8.06 23.04 10.68 
Benchmark Early 14 35.79 11.72 47.21 11.38 50.62 14.28 11.43 13.73 14.77 16.13 
Late 12 38.17 7.60 38.50 11.09 53.67 13.36 0.33 6.61 15.5 10.79 


Note. Comp = Composite, MC = Magnitude Comparison, MN = Missing Number, NI = Number 


Identification 
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Table 2. 


Average KinderTEK usage by classroom for each instructional mode and overall 


Start Class KinderTEK 4M total M cal. M days of M total Mlessons’ M different 
type (n) mode?* daysof | weeksof usepercal. minutes of seen in lessons 
use use week instruction? mode seen 
Early Class A Sequenced 20.71 - - 247.47 20.57 - 
(n= 21) 
OVERALL 20.71 25.98 0.79 247.47 - 20.57 
Early ClassB Sequenced 31.50 - - 374.92 18.94 - 
(n= 18) 
OVERALL 31.50 21.94 1.45 374.92 - 18.94 
Early ClassC Sequenced 25.11 - - 270.13 23.79 - 
(re) a8 
Directed 6.21 - - 51.72 5.79 - 
Exploration 2.26 - - 16.79 8.16 - 
OVERALL 33.42 25.11 1.34 338.64 - 29.63 
Late Class D Sequenced 37.93 - - 460.43 31.93 - 
(n= 15) 
OVERALL 37.93 12.78 2.97 460.43 - 31.93 
Late Class E Sequenced 22.12 - - 230.25 24.00 - 
(n= 17) 
OVERALL 22.12 12.52 1.77 230.25 - 24.00 
Late Class F Sequenced 12.09 - - 136.90 13.64 - 
(n = 22) ee 
Screening® 0.14 - - 0.94 0.36 - 
OVERALL 12.18 9.53 1.53 137.84 - 13.64 


“Sequenced mode presents the KinderTEK curriculum such that students learn prerequisites 
before encountering more advanced content. Sessions are timed (e.g., 15 minutes). Directed 
mode allows teachers to select one to two content areas that are presented in a sequenced fashion 


in timed sessions. If students master the activities in both the assigned content areas, they review 
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activities until the teacher makes a change. During this study, one teacher activated directed 
mode and selected “story problems” (which appear early in sequenced mode) and 
“decomposition of teens” (which are the last lessons in sequenced mode). Exploration mode is 
untimed, and students have control over what to work on (e.g., any math-focused lesson or 
reward activity) and for how long; they can exit lessons at any time. Screening mode is a non- 
instructional, timed mode, that consists of just pretests for each activity. No feedback is 
provided. 

Minutes of instruction refers to time spent in instructional activities (i.e., excluding transitions 
and reward activities, which are part of the 15-minute structured experience). 


‘Only one student in 


Page 45 of 49 


LEARNING GAINS FROM THE KinderTEK iPAD MATH PROGRAM 


Table 3. 


Results of Tl to T2 Gain Score Analyses 


Effect test Dp BH Adj p BH critical Hedges’ g 
value [95% CI] 

ASPENS 5.503 021 057 025 0.44 [0.06, 0.81] 
Composite 
ASPENS 4.791 029 057 050 0.33 [-0.04, 0.70] 
Magnitude 
Comparison 
ASPENS Missing 4.070 .046 061 .075 0.38 [0.00, 0.75] 
Number 
ASPENS Number 2.919 .090 .090 .100 0.32 [-0.05, 0.69] 


Identification 


Note. BH Adj p = Adjusted p value using Benjamini and Hochberg's (1995) correction for false 


discovery; BH critical value = value against which to compare the BH Adj p allowing for a 


false discovery rate of .10; CI = Confidence Interval. Control n = 58 and treatment n = 56. 
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Figure | 
Study Design 
T1 Fall T2 Winter & Spring T3 
assessment Use of assessment Use of assessment 
KinderTEK KinderTEK 
Treatment xX xX xX xX xX 
Comparison xX XxX Xx Xx 


RQ! - KinderTEK vs. BAU 


RQ2 - Early Start vs. Late Start (& exploration of differences by initial skill) 
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Figure 2. 


Early start vs. late start student math performance at T1, T2, and T3 for all students by condition 
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Figure 3. 
Early start vs. late start student math performance at T1, T2, and T3 for ASPENS by ASPENS 
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